This invention relates generally to computer systems, and more particularly to methods of test data generation in a computer system.
As it is known in the art, testing, particularly as applied to computer hardware and software, is an ongoing task in development and maintenance activities which requires a large portion of time, computer and human resources, and effort. Software development may include the development of an entirely new application or program, or the addition of a new feature to an existing application. Software maintenance activities generally include the correction of reported problems. Similarly, for example, hardware development may include the development of a new computer chip, or an improvement to an existing computer chip.
Testing is typically performed to verify proper functioning of new hardware and software, as well as to verify correctness of a modification or an improvement to existing hardware and software.
Typically, data, such as test data, is used as an input during testing of hardware and software to provide, for example, sample inputs of real-time conditions under which the hardware or software functions to enable problem detection and resolution. During testing, a developer or maintainer typically attempts to debug hardware or software by detecting and solving problems associated with a product prior to the product, for example, reaching a user or customer.
Generally, as the complexity of hardware and software increases, the complexity of testing also increases. For example, as the design of a computer chip increases in complexity, the number and combination of test cases needed to ensure that each portion of the computer chip functions correctly may also increase.
Adequately and thoroughly testing hardware and software is typically an important aspect of development and maintenance. In testing, it is important that the test data provide thorough testing coverage, for example, of each component and many or all combinations of components, to adequately test combinations of the components as they may be used outside of the testing environment.
One technique used to generate test data includes manually writing each data case. For example, each data case may be manually entered into a file in a computer system using an input device such as a keyboard connected to a terminal.
This technique has several drawbacks. One drawback is that when such data entry is performed manually, it is often cumbersome and time-consuming. Also, possibility of a data entry error causing incorrect data to be entered and used, generally increases with manual data entry. Another drawback is that a human typically constructs each data case. This often results in incomplete testing coverage since a combination may easily be omitted. Particularly as the complexity of the hardware or software being tested increases, the probability that a necessary test case is omitted also increases since the various combinations of elements to be tested increases. In summary, manual data entry is typically costly, inefficient, and prone to human error.
Another method of generating test data includes using a tool, such as a software tool, to automatically generate test data. This is an improvement over the foregoing manual data entry technique since it is more efficient and less prone to human error due to the decreased requirement of human interaction. However, current tools used for this purpose have drawbacks and often lack desirable features and functions.
Generally, tools currently available use independent test data generation methods, such as `random` and `exhaustive`, that are associated with a single input data set providing independent data element selection, i.e., select a data element from one set independently of other data elements selected from other input data sets. For example, a `random` selection method randomly selects, using a random number generator selection algorithm, a first data element in a first input data set to be included in a generated test data case. Additionally, using a second input data set with the `random` selection method causes a random selection of a second data element from the second input data set to be output in the same generated test data case. An `exhaustive` method of test data generation may, for a single set, alternate through each data element. Once each data element has been selected for test data generation, selection begins again with the first data element cycling through to the last data element.
Current methods of generating data frequently produce fixed sequences of data which may also omit certain combinations of data elements in a single data case. It is desirable to have a method for generating data which does not inherently omit combinations of input data elements in its generated sequence of test cases. Furthermore, the ability to alter a sequence and produce a different ordering of test cases for the same input data elements is also desirable. Altering a sequence for a given set of test elements may provide a new additional testing perspective for the same input test elements thereby increasing test coverage without requiring respecification of input data elements.
Existing methods tend to be independent, generating test data that focuses on a single test case. Generally, thorough testing also includes using test data which coordinates a particular sequence of test cases, not just the combination of test elements in each isolated test case. For example, testing retrieval of information from a data memory cache requires a series of test cases in which a first instruction including an address is near a second instruction also including the address. Therefore, a method of generating test data which focuses on coordinating generated test data cases is desirable.
In testing hardware and software, exhaustive testing may not be possible or feasible. The method should be able to generate data that provides adequate testing coverage using an amount of generated data that can be executed in a reasonable amount of testing time.